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Abstract 
Executive function (EP), critical for many developmental outcomes, emerge in childhood and 


continue developing into early adulthood (Blakemore & Choudhury, 2006). During adolescence 
there are important developments in Hot EF, which involves using EF in emotionally salient 
contexts (Zelazo & Carlson, 2012). The current study used an emotional design approach (Um, 
Plass, Hayward, & Homer, 2012) to study hot EF in adolescents playing a digital game designed 
to train the EF subskill of shifting. Participants (NV = 233; aged 12 to 16, M = 13.8, SD = 1.1) 
played one of two version of the game: hot (with emotional design), or cool (with more 
emotionally neutral design). There was a main effect of condition, with higher posttest scores on 
shifting in the hot condition. Condition significantly interacted with age: older adolescents had 
better outcomes in the hot condition than in the cool condition. A three-way interaction between 
age, prior EF and condition was found, indicating that the age by condition interaction was 
affected by prior EF. These results indicate that the higher emotional arousal in the hot condition 
is more effective for enhancing EF skills, particularly for older adolescents, and argue for 
developing learning and training games that account for developmental changes such as the 


growth of hot EF in adolescence. 


Activating Adolescents’ “Hot” Executive Functions in a Digital Game to Train Cognitive Skills: 
The Effects of Age and Prior Abilities 

Executive Function (EF) refers to a set of related cognitive processes that allow for the 
planning, monitoring and control of cognition, behavior and emotions in order to remain goal- 
oriented (Best & Miller, 2010; Diamond & Lee, 2011). EF first emerges in early childhood and 
continue to develop throughout adolescence into early adulthood (Blakemore & Choudhury, 
2006). Although tied to specific brain developments, environmental and experiential factors have 
been shown to have a great influence on the development of EF. For example, the stressors of 
poverty have been associated decreased EF in preschool children (Blair et al., 2011; Blair & 
Razza, 2008; Sarsour et al., 2011). These disparities in early EF may contribute to the 
achievement gap that continues to widen over time (Finn et al., 2016; Noble, McCandliss, & 
Farah, 2007; Ryan, Fauth, & Brooks-Gunn, 2006; Welsh et al., 2010). Conversely, numerous 
approaches have been identified that support the development of EF, including explicit 
classroom curricula, martial arts training, and digital interventions (Diamond & Lee, 2011). 
Interest in finding ways to improve EF is driven by the considerable evidence of the importance 
of EF for a number of behavioral and developmental outcomes in children and adolescents (Blair 
& Razza, 2007; Miller & Hinshaw, 2010; Welsh, Nix, Blair, Bierman, & Nelson, 2010). 

Most successful EF interventions share two common features: repeated practice and 
activities that become progressively more challenging (Diamond & Lee, 2011). Even though 
computerized interventions, including digital games, have both of these features, evidence for the 
effectiveness these types of interventions has been mixed. Although some reviews have found 
that digital games have had limited success in training EF (e.g., Mayer, 2014; Powers, Brooks, 
Aldrich, Palladino, & Alfieri, 2013), other recent studies have found that some digital games can 


be effective tools for supporting the development of EF (Anguera et al., 2013; Homer, Plass, 


Raffaele, Ober, & Ali, 2018; Parong et al., 2017). Given that only some digital trainings of EF 
are effective while others are not, it is of both practical and theoretical importance to identify the 
specific features that can make digital games effective tools for learning and cognitive skills 
training (Plass, Homer, & Kinzer, 2015). 

The goal of the current study is to examine the effectiveness of developmentally 
informed features of a game created to support the development of EF in adolescence. The 
premise is that matching design features to relevant developmental factors will help make a 
digital game-based intervention more effective. Specifically, in consideration of developments in 
Hot EF (Zelazo & Carlson, 2012) that occur during adolescence, we investigate the use of an 
emotional design approach (Plass, Heidig, Hayward, Homer, & Um, 2014) to make a digital 
game intended to train the EF subskill of shifting more appropriate for older versus younger 
adolescents. To understand best how to create a developmentally appropriate game design for 
adolescents requires first considering the developmental trajectory of EF, including relevant 


neurophysiological changes. 


The Development of Executive Functions 


Although there is variation in how the specific components of EF are characterized, one 
of the most widely accepted approaches is Miyake and colleagues’ Unity and Diversity 
framework (Miyake et al., 2000). In this framework, EF involves sub-skills that are related to 
one another (i.e., unified) but are also statistically distinct, differentially predict performance on 
cognitive tasks, and manifest at different developmental periods (i.e., diverse). Subskills include 
inhibition, the ability to suppress behavioral or cognitive impulses, shifting (or switching), the 
ability to mentally shift attention between tasks, and updating (or working memory), the ability to 


hold and manipulate information in one’s mind (Miyake et al., 2000). The EF components 


support development of more complex higher-order skills, such as planning and reasoning 
(Diamond, 2013; Ortner, Kilner, & Zelazo, 2007; Zelazo & Lyons, 2011) 

EF first clearly emerges around 24 months of age, with additional rapid developmental 
change between the ages of 3 to 6 (Carlson, 2005; Jones, Rothbart, & Posner, 2003), and clear 
evidence of continued growth well into adolescence (Hughes, 2011). Although ages at which the 
subskills of EF reach adult levels are not absolutely determined, in part because of variation in 
performance on the specific tasks used to assess EF (see Best & Miller, 2010; Perone, Almy, & 
Zelazo, 2018), the pattern of development is fairly consistent. Inhibition appears to develop first 
and continues to develop in adolescence, with shifting and working memory developing later and 
not reaching a mature adult state until late adolescence or early adulthood (Davidson, Amso, 
Anderson, & Diamond, 2006; Huizinga & van der Molen, 2007). These developments may be 
reflected in behavioral difference. For example, when solving problems, younger children tend to 
rely more on inhibition, while older children and adolescents rely more on shifting and working 
memory (Davidson et al., 2006; Diamond, 2013; Huizinga, Dolan, & van der Molen, 2006; 
Isquith, Crawford, Espy, & Giola, 2005). 

Most broadly, EF has been associated with activation in the lateral prefrontal cortex 
(PFC) within the cerebral cortex (Kane & Engle, 2002; Zelazo & Miiller, 2002). During 
adolescence, beginning with the onset of puberty, there is an accelerated period of EF 
development that is marked by a linear growth in the amount of white matter in the frontal areas 
of the brain, likely reflecting an increase in neuronal myelination. At approximately the same 
time, the amount of gray matter changes with age in an inverted U-shaped pattern, reflecting 
synaptic pruning and further reorganization (Barnea-Goraly et al., 2005; Blakemore & 


Choudhury, 2006; Sowell et al., 2004). MRI studies have shown that gray matter may continue 


to steadily increase until puberty and then proceed to decrease from adolescence through early 
adulthood (Sowell et al., 2003; Sowell et al., 2001). The increased myelination during early 
adolescence results in increased speed of neuronal transmission, and the decrease in gray matter 
from synaptic pruning during later adolescence results in improved speed and efficiency of 
cognitive processing, including improvements in adolescents’ EF (Best, Miller, & Jones, 2009; 
Davidson et al., 2006; Huizinga et al., 2006; Luna & Sweeney, 2004). 

Although adolescents reach mature, adult-like levels of performance on some standard 
EFF tasks (Huizinga et al., 2006), behaviors associated with impulsivity and risk-taking continue 
to be observed, especially in contexts that elicit an affective or emotional response (Leshem, 
2016), such as when faced with peer pressure (Blakemore & Robbins, 2012). Researchers have 
attributed such conflicting behaviors (i.e., demonstrating mature EF skills in combination with 
risky-decision making) to the early development of subcortical regions responsible for emotions 
and motivation and the later development of the PFC (Blakemore & Robbins, 2012; Dahl, 2004). 
Further, the lack of impulse control and higher levels of risk-taking in adolescents may be 
attributed to the discrepant time-course of development of these brain structures (Steinberg, 
2008). The ability to execute cognitive control in conditions of high motivation and emotion has 
been identified as “hot EF”, which undergoes important developments during adolescence 


(Zelazo & Miiller, 2002). 


Hot EF and Cool EF Development 


Although most EF research has focused on lab-based performance in emotionally neutral 
conditions, cognition is often different in emotionally laden situations (Bechara, 2004). To 
delineate EF processes in motivationally significant situations, and to identify reasons why 


adolescents’ decisions do not consistently align with their objectives, researchers have 


distinguished between “hot EF” and “cool EF” (Perone et al., 2018; Welsh & Peterson, 2014; 
Zelazo & Miiller, 2002). Cool EF refers to the top-down processes that are utilized in 
decontextualized and abstract situations that lack clear rewards, punishments or personal 
consequences. Measurement of such cool EF involves abstract, decontextualized stimuli such as 
those found as the flanker, n-back and Dimensional Change Card Sort (DCCS) tasks, in which 
participants are required to inhibit responding to arrows, numbers, or neutral shapes and colors. 
Hot EF refers to the top-down processes that are elicited in emotionally salient real-world 
contexts marked by clear rewards and personal consequences. Measurements of hot EF, such as 
the Iowa Gambling Task, are designed to evoke an emotional response (Hongwanishkul, 
Happaney, Lee, & Zelazo, 2005; Zelazo & Carlson, 2012) by having “real world” consequences, 
such as participants winning or losing real items (1.e., money or candies). Hot EF is also involved 
in situations that require affect regulation, reflective response generation, and management of 
motivation and emotions (Perone et al., 2018; Prencipe et al., 2011). It should be noted that “hot” 
and “cool” EF are not distinct processes but rather represent similar basic cognitive functions 
that fall along a continuum of emotional activation (Perone et al., 2018). 

Past research has sought to pinpoint the factors and developmental trajectories underlying 
hot and cool EF by analyzing differences in outcomes on hot EF tasks and cool EF tasks. 
Neurophysiological studies suggest that although there is significant overlap in the neural 
networks involved, there are still difference in the brain areas recruited for hot and cool EF. The 
dorsal and lateral aspects of the PFC have been shown to be associated with cool EF. For 
example, during completion of the DCCS (measure of task-switching or shifting), activation in 
the anterior cingulate cortex (ACC) occurs as soon as participants notice the conflict that is 


inherent in the rule change. This activation of the ACC catalyzes recruitment of the lateral PFC 


regions (Botvinick, Braver, Barch, Carter, & Cohen, 2001; Ezekiel, Bosma, & Morton, 2013). 
Ventro-medial regions of the PFC (including the orbitofrontal cortex), which are connected to 
the limbic system and are responsible for cognition and emotion integration, have been 
associated with hot EF (Zelazo & Carlson, 2012; Zelazo & Miiller, 2002). Lesion studies have 
indicated that hot and cool EF may be dissociative, as patients with damage to the orbitofrontal 
cortex experience impairments on hot FF tasks but still perform well on measures of cool EF 
(Bechara, Damasio, Tranel, & Anderson, 1998). Even though different areas of the brain may be 
responsible for hot and cool EF, they do appear to be part of a coordinated system. Situations and 
contexts that differentially elicit hot or cool EF processes, seem to fall on a continuum that 
requires increasingly complex and developed abilities (Perone et al., 2018; Prencipe et al., 2011). 

Research on developmental trajectories of cool and hot EF from childhood through 
adolescence supports the continuum viewpoint. While age-related improvements in cool EF are 
found in childhood (ages 8-13), improvements in hot EF do not occur until adolescence, typically 
starting around age 14 (Crone & van der Molen, 2004; Hooper, Luciana, Conklin, & Yarger, 
2004; Poon, 2018; Prencipe et al., 2011). It should also be noted that adolescents are often not 
performing at ceiling on measures of hot EF suggesting that it is still developing throughout 
adolescence (e.g., Prencipe et al., 2011; Hooper et al., 2004). Poon (2018) also argues that while 
cool EF follows a linear trend across development, hot EF follows a bell-shaped curve (as 
measured by increases in risk-taking), resulting in a vulnerability to risky-decision making 
during middle adolescence (ages 14-15). Steinberg (2008) analyzed age-related differences on 
delayed discounting and future orientation and found that participants who were 16-year-olds 
and older were more likely than children between the ages of 10-16 to delay an immediate 


smaller reward for a future larger reward. 


Overall, the protracted development of hot EF from childhood through adolescence may 
denote a critical or “sensitive” period of brain plasticity during middle adolescence (Zelazo & 
Carlson, 2012). These periods of vulnerability or sensitivity are also opportunities for creating 


effective and developmentally appropriate cognitive training tools. 


Emotional Design and Learning 


Typically, tasks involving hot EF, such as the lowa gambling task, use external rewards 
to evoke emotions. In real life, a wide variety of experiences can evoke hot EF including dealing 
with peer pressure, social exclusion, experiencing a delay of gratification, as well as dealing with 
potential gains or losses. In other words, hot EF is involved in situations that are not only 
motivationally significant, but also emotionally significant. This means that another option for 
evoking hot EF in the lab is by using materials that evoke emotions themselves. 

An Emotional Design approach (Plass et al., 2014; Plass & Kaplan, 2016; Um et al., 
2012) is one method of evoking emotions through the design of materials. Designers of digital 
games have long recognized the importance of emotional responses, and have identify various 
game features that can be used to induce emotions (Plass et al., 2015). Past research on the 
effects of visual design (e.g., shapes and colors) has shown that bright, vivid and saturated colors 
can positively affect emotions and feelings (Gao & Xin, 2006; Tucker, 1987; Valdez & 
Mehrabian, 1994). Shape of design elements can also induce emotions. Examples include the 
baby-face bias (Lorenz & Generale, 1960), and anthropomorphism (Hongpaisanwiwat & Lewis, 
2003). The baby-face bias refers to the extent to which round shapes, short chins, small noses 
and large eyes are perceived as baby-like features that induce positive affect over sharped edged 
shapes (Lorenz & Generale, 1960). Anthropomorphism concerns the tendency for humans to 


attribute human-like features and characteristics to inanimate objects. For example, animated 


game characters expressing human-like emotions can serve to elicit and sustain attention and 
engagement in the learner (Hongpaisanwiwat & Lewis, 2003). 

Based on the emotional effects of design features, there has been interest in finding the 
extent to which specific colors and shapes within multimedia environments can promote learning 
through elicitation of positive emotions (Mayer & Estrella, 2014; Plass et al., 2014; Um et al., 
2012). Um et al. (2012) designed two types of multimedia learning environments on 
immunization that were identical in terms of their informational content, but differed in their use 
of shapes and colors. The researchers also examined if internally induced emotions (i.e., 
emotions evoked through multimedia design), were more sustainable than externally induced 
emotions (i.e., emotions evoked using a standard mood-induction paradigm). The positive 
emotional design environment included round shapes and vivid colors, while the neutral 
emotional design environment included simple square or circle shapes and neutral colors (grays). 
Um et al. (2012) found that round shapes and vivid colors induced more positive emotions than 
the neutral shapes and colors. More importantly, learners in the positive emotional design group 
scored higher on comprehension and transfer measures than did those in the neutral group. 
Additionally, internally induced emotions were sustained until the end of the learning task, 
unlike externally induced emotion, which faded before the end of the task. Further, internally 
induced emotions had a greater effect on comprehension and transfer. These, and subsequent 
findings (Mayer & Estrella, 2014; Park, Knoérzera, Plass, & Briinkena, 2015; Plass et al., 2014), 
have clearly identified emotional design as an approach that can enhance learning without 
directly changing the structure or content of the learning environment. 

The ability for visual features of game characters to elicit emotions in the learner was 


further examined by Plass and colleagues (MacNamara et al., 2018; Plass et al., 2018). The 
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researchers were interested in the extent to which four different visual features of game 
characters (shape, color, expression, and dimensionality) influenced the emotions participants 
attributed to game characters. In a series of forced-choice trials, participants viewed two images 
of game characters side by side, that varied in shape (round or square), dimensionality (2D or 3D 
with shading), color (red or grayscale), and expression (happy/sad or neutral), and were required 
to choose which images made them feel the happiest or saddest. Overall, for both emotions, there 
was a large effect of expression, and a medium effect of color and shape, with little to no effect 
of dimensionality. These findings systematically identify features that can be used for emotional 
design in multimedia for learning or cognitive training, including the design of digital games 


intended to support the development of EF. 


Improving EF through Game-based Training 


There has been great interest in identifying ways to systematically trained and improved 
EF (Best & Miller, 2010; Blair & Razza, 2007; Diamond & Lee, 2011). Interventions based on 
digital games have been one method to enhance EF skills such as inhibition, working memory, 
and shifting (Anguera et al., 2013; Boot, Kramer, Simons, Fabiani, & Gratton, 2008; Homer, 
Plass, et al., 2018; Parong et al., 2017; Strobach, Frensch, & Schubert, 2012). The effectiveness 
of games for training EF can be attributed to a number of factors, including their ability to 
preserve engagement and motivation, adapt to the player’s current challenge level, and ability to 
promote distributed practice (Diamond & Lee, 2011; Dye & Bavelier, 2010; Holmes, Gathercole, 
& Dunning, 2009; Homer, Ober, & Flynn, 2018). 

Currently, few if any game-based interventions explicitly target hot EF, although games 
would seem to be a viable medium for training this ability. Surprisingly, most of the research on 


game-based trainings of EF has focused on young children, college-aged students and aging 
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adults (Best et al., 2009), with a paucity of research with adolescents, even though this is a 
period of growth in EF, particularly, as described above, growth in hot EF (Alloway, 2012; 
Green et al., 2012; Roughan & Hadwin, 2011; Zinke, Einert, Pfennig, & Kliegel, 2012). 
Additionally, although there is considerable research on the effectiveness of training working 
memory, there is much less work on training of other EF skills, such as shifting, even though the 
importance of shifting for outcomes such as academic success has been well established (Homer 
& Plass, 2014; Yeniad, Malda, Mesman, van Ijzendoorn, & Pieper, 2013). As stated above, 
shifting, or cognitive flexibility, requires the ability to switch from adhering to one rule to 
another with minimal perseveration and increasing flexibility. It emerges in early childhood and 
continues to develop into adolescence. Deficits in shifting have been associated with a number of 
neuro-cognitive disorders, including autism (e.g., Ozonoff, Pennington, & Rogers, 1991), and 
ADHD (e.g., Willcutt, Doyle, Nigg, Faraone, & Pennington, 2005). 

There have been a few successful efforts to train shifting in adolescents. For example, 
Homer, Plass, et al., (2018) asked high school students to play a digital game designed to 
improve the EF subskill of switching at least once a week over the course of 6 weeks. High 
school students who played the game demonstrated significant pre- to post gains in shifting (as 
measured by the DCCS), as well as near transfer to inhibition (as measured by the Flanker task). 
Similarly, Parong et al. (2017), established the validity of an adaptive game-based training 
targeting shifting called All You Can E.T. (AYCET). The researchers found that late-adolescent 
college students who completed at least 2 hours of game-play over multiple sessions displayed 
greater improvements on two measures of cognitive flexibility or shifting (i.e., the DCCS and 
letters and numbers task) than did active controls who played a word search game. A second 


experiment confirmed these results by comparing those in the experimental group to an inactive 
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control group. These studies provide evidence that it is possible to improve the EF subskill of 
shifting through a digital game-based intervention, particularly with a game that was developed 


with the explicit intention of training EF. 


Current Study 


The current study investigates how the design of a game for training EF in adolescents 
(aged 12-16) can be enhanced by considering the neurophysiological and cognitive changes 
related to EF occurring during this developmental period. Specifically, an emotional design 
approach (Plass et al., 2014) was used to make a “hot” and “cool” version of All You Can E.T. 
(AYCET; Parong et al., 2017), a digital game designed to train the EF subskill of shifting. The 
hot version of AYCET was created by altering attributes of the game characters by using bright 
colors and emotional expressions, and a cool version was created by using characters with more 
emotional neutral attributes, including gray-scale coloring and neutral expressions. Given that 
hot EF undergoes increased development beginning at mid-adolescence (Principe et al., 2011; 
Zelazo & Carlson, 2012), it was hypothesized that the hot version of the game would be 
particularly beneficial for adolescents starting around age 15. Finally, for the purpose of the 
current research, we only examined the effects of training over a 20 minutes period. Although 
sustained EF improvements most likely require a more prolonged intervention, some of our prior 
research has shown that small, but significant gains in EF can be observed even after a brief 
intervention (e.g., Homer, Ober, Rose, et al., 2018). 

For the current study, we posed four related research questions: 
Research Question 1: Will there be significant changes in EF after playing AYCET? 

a) Following 20-minutes of gameplay of AYCET, will there be significant changes in 


participants’ EF subskill of shifting as indicted by posttest scores on the DCCS? 
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b) Will any pre- to posttest differences in shifting vary between the two conditions? 
We hypothesize a) significant gains in shifting from pretest to posttest, and b) 
significantly greater gains in the hot condition. 
Research Question 2: Will age and AYCET condition interact to affect adolescents’ 
improvements of EF? 
Given that hot EF develops later in adolescence, we hypothesize that the benefits of the 
hot condition of AYCET would be greater for the older adolescents. 
Research Question 3: Do differences in game-based performance predict posttest scores in 
the EF subskill of shifting, as indicted the DCCS? 
We hypothesize that better game metrics will predict better posttest scores in measures of 
EF. 
Research Question 4: Are there significant differences between the two AYCET conditions 
in game-based performance metrics (i.e., measures of speed and accuracy)? 
We hypothesize that participants in the hot condition of AYCET will demonstrate 


significantly better speed and accuracy during gameplay. 


Methods 


Participants 


Participants (N = 233) were 12- to 16-year-old adolescents (M = 14.03, SD = 1.33) from 
public schools in New York City. Approximately 44% of the students identified as female, 56% 
identified as male, with one of the adolescents declining to self-identify gender. Of those 
participants who indicated their ethnicity, the majority self-identified as Hispanic/Latino 
(90.1%), with the remainder self-identifying as Asian or Pacific Islander (4.3%), 


white/Caucasian (2.8%), and African American (2.8%). As an indication of SES, participants 
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were asked about their mother’s highest level of education, the following was reported: 5% had 
completed up until at least 8th grade but had not completed high school; 14% had some high 
school education but had not completed a degree; 16% had completed high school; 17% had 
some college; 17% had a college degree; and 6% had some graduate school experience. The 
remainder of participants either did not respond or indicated they did not know their mother’s 


highest level of education (15.0%). 


Materials 


All You Can E.T. (AYCET). 

All You Can E.T. (AYCET) is an adaptive digital-game designed to train the EF subskill 
of shifting. AYCET was created by game developers and researchers at the CREATE lab at 
NYU. As mentioned above, interventions using earlier versions of this game have been shown to 
significantly improve adolescences’ EF subskill of shifting (e.g., Homer, Plass, et al., 2018; 
Parong et al., 2017), with some increase (as indicated by pre- to posttest gains in scores on the 
DCCS) found even after just 20 minutes of gameplay (e.g., Homer, Ober, Rose, et al., 2018). 

The objective of AYCET is to give aliens the appropriate food or drink. The game is 
governed by rules, with rule changes occurring within each level and between levels. During 
AYCET, the player is required to give different types of aliens (e.g., one-eye vs. two-eye; red vs. 
blue) either food or drink. Correct responses depend on the relevant rules that are presented to 
the player before or during each level. For example, a rule may require the player to give food to 
two-eyed aliens and a drink to one-eyed aliens. A rule change would then let the player know 
that they need to now allocate a food to one-eyed aliens and drink to two-eyed aliens. The game 
interface notifies players of rules and rule changes. As levels increase, the rules and rule-changes 


become more complex (e.g., one-eyed red aliens and two-eyed blue aliens need food; two-eyed- 
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red aliens and one-eyed blue aliens need drink). The complete game has a total of 16 levels, but 
with only 20 minutes of gameplay, participants in the current study would not have enough time 
to complete all levels. AYCET is adaptive in that it changes the speed of oncoming aliens based 
on the player’s performance; speeding up if the player is accurate and responding quickly, 
slowing down if the player is making errors or is taking a long time to respond. Due to its 
adaptive nature, players may experience different amounts game levels and rule changes. Game 
play lasted for 20 minutes. 

For the current study, two versions of AYCET were created: a “hot” version and a “cool” 
version. The hot version was designed to more induce greater emotions in players. (See Figure 
1.) This version featured round-shaped aliens that were vividly colored (i.e., red and blue). When 
a player makes a correct hit (e.g., when they correctly give blue one-eyed aliens a drink) the alien 
goes from expressing a neutral emotion to a happy emotion (i.e., smiles). When the player makes 
an incorrect hit (e.g., when they incorrectly give blue one-eyed aliens food) the alien goes from 


showing a neutral expression to a sad frowning expression. 
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T/ of 


Figure I. Screen capture of the hot EF condition showing alien’s emotionally expressive features 
and vivid color palette. 

The cool version was identical to the hot version except the Aliens were designed to not 
induce as strong of an emotional response in players. (See Figure 2.) This version featured 
square-shaped aliens that were colored using a grey-scale pallet. Furthermore, the aliens did not 
express joy or sadness when fed correctly or incorrect, although feedback was given to the 


players. 
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Figure 2. Screen capture of the cool EF condition showing alien’s neutral emotional expressive 
features and a grayscale color pallet. 
Game Log Data. 

AYCET game log data was recorded through DREAM integrated research platform while 
participants played one of the two versions of the game. Metrics from the log data were used as 
indicators of performance. Specifically, as a measure of accuracy, the proportion of targets 
correctly hit over the total number of presented targets was calculated for each level. As a 
measure of speed, the average reaction time for targets correctly hit for each level was recorded. 
Demographic Survey & Postgame Questionnaire. 

Participants received a brief demographic questionnaire. This questionnaire asked about 
gender, age, race, and parental education history. After completing the game, participants were 
given a brief, 3-question survey about their feelings for the aliens in the game. Participants were 


asked to rate on a 7-point Likert scale the extent to which: 1) they were happy when they helped 
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the aliens; 2) they were sad when they didn’t help the aliens; and 3) they cared about what 
happened to the aliens. A total “Emotional Engagement” score was then calculated by summing 
scores on the three questions. 

Dimensional Change Card Sort Task (DCCS). 

Designed by Zelazo and colleagues (Zelazo, 2006; Zelazo et al., 2013), the DCCS is used 
to measure cognitive flexibility or task shifting. Although the DCCS has been used most 
commonly with children, the version used in the current study, which was based on the 
computerized version of the DCCS included in the NIH Toolbox Cognition Battery (Zelazo et 
al., 2013), has been validated for ages 3-85. This version of the DCCS has strong test-retest 
reliability (ICC=.92) and convergent and discriminant validity (Zelazo et al., 2013). 

The computerized version of the DCCS consists of practice and test blocks. During the 
practice blocks, participants are shown two target stimuli on either side of the screen and asked 
to match a stimulus presented in the middle of the screen to one of the two target stimuli 
depending on a rule. For practice and test blocks, the rule appears on the screen between each 
trial. For the practice block, participants are provided with feedback concerning their correct or 
incorrect responses. In the practice block, participants were instructed to match bivalent test 
stimuli to target stimuli (e.g. white rabbit and a green boat) according to the rule (i.e., shape or 
color). In the test blocks, participants were presented with pictures that varied across two 
dimensions: 1) shape (e.g., baseball versus sailboat), and 2) color (e.g., blue versus yellow). 


Performance was measured by a combination of speed and the number of accurate trials. 


Procedure 


Participants were recruited through partnerships with schools in New York City. Parental 


consent was obtained before the students participated in the study. Testing took place in class- 
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groups at a university-based research lab. When arriving at the lab, the students were 
individually randomly assigned study ID numbers, which had previously been associated with 
access codes corresponding to either the hot or cool version of AYCET. In smaller groups, the 
participants would sit down at the lab computers and enter their ID and access codes into the 
appropriate field of the Digital Reference, Experiment, and Assessment Manager (DREAM; 
CREATE, 2016), a web-based platform that enables researchers to upload study contents and 
allows participants to access contents using access codes. In DREAM, participants assented to 
participant in the study, and then completed the following tasks in sequential order: DCCS 
pretest, 20 minutes of AYCET Hot or Cool Version, Emotional Game Characteristics 
Questionnaire, DCCS posttest and Demographics and GamePlay Questionnaire. Game log data 
was continuously collected while the player completed AYCET. Research assistants were 
available during the study to answer questions but provided minimal assistance to participants. 


After completing the session, participants were debriefed in a group session. 


Results 


Data was collected through the DREAM system, an integrated research and digital media 
delivery platform. Performance on the DCCS pretest score was examined to identify outliers 
following Tukey’s method (Tukey, 1977), with the modification that instead of removing outliers 
beyond 1.5 IQR, we removed values beyond 3 IQR from the first and third quartiles. In data that 
involves reaction times and is derived from a developmental sample, greater variability is to be 
expected, and so this decision made to allow for the greater variability inherent in the sample. In 
total, 3 participants were removed from the sample, all of whom had pretest scores that were 


significantly below the overall average. This left a final sample size of 233, with 126 who had 
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been randomly assigned to the cool condition of AYCET and 107 who had been assigned to the 
hot condition. 

Initial analyses examined the Emotional Engagement score, which was calculate by 
summing responses to the three post-game questions about how much participants cared about 
the aliens in AYCET. A significant difference was found between the emotional engagement 
score for the two conditions, ¢ (230) = 2.34, p = .02, D = .30, with participants in the emotionally 
hot condition (M = 10.2, SD = 2.8) indicating more emotional investment in the aliens than 
participants in the cool condition (M = 9.3, SD = 3.2). However, participants’ emotional 
engagement score was not related to posttest DCCS scores, r = .07, p = .28, and so the variable 
was not included in subsequent analyses. Similarly, possible effects of gender on DCCS posttest 
scores was examined, and no significant gender differences were found, f (230) = .33, p = .74. 
Therefore, gender also was not included as a factor in the subsequent analyses. 

Effects of Condition and Prior-EF on Posttest DCCS Scores 

To determine if there was an overall gain in shifting as measured by changes pre- to 
posttest on the DCCS, a paired sample t-test was conducted. Results indicated that there was a 
significant difference, t (232) = 4.10, p < 0.001, d= 0.27, with average DCCS posttest score (MV 
= 7.25, SD = 1.38) being significantly higher than average DCCS pretest score (M = 6.80; SD = 
1.70). Further analyses were then conducted to examine the effects of condition and age on 
posttest DCCS scores. 

To examine the effects of age and condition on shifting, an ANCOVA was conducted 
with DCCS posttest score as the dependent variable, condition (hot vs. cool) as an independent 
factor, and DCCS pretest score and age as covariates. Results indicated a significant main effect 


of condition, F(1, 225) = 4.91, p = .03, partial eta squared = .02. Examination of the adjusted 
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marginal means indicated that the hot condition had significantly greater average posttest scores 
(M = 7.33, SE = 1.22) than did the cool condition (M = 7.17, SE = .11). There was also a 
significant main effect of age, F(1, 225) = 4.15, p = .04, partial eta squared = .02. The main 
effects were mitigated by a significant age by condition interaction, F(1, 225) = 4.96, p = .03, 
partial eta squared = .02, indicating that the effects of condition were different for different aged 
participants. To further examine the interaction between condition and age, a DCCS change 
score was created by calculating the difference between posttest and pretest scores on the DCCS. 
As illustrated in Figure 3, for the youngest participants difference in pre- to posttest change in 
DCCS was about the same for both the hot and cool conditions. In contrast, the older participants 


showed significantly greater gains in the hot condition than in the cool condition. 
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Figure 3. Pre- to Posttest Gains in Shifting (DCCS) by Age Group for Hot and Cool AYCET 
Conditions 

A significant condition by pre-DCCS score interaction was also found, F(1, 225) = 4.11, 
p = .04, partial eta squared = .02, indicating that the effects of condition were different for 
participants with different initial pre-DCCS scores. As seen in Figure 4, there was little 
difference between conditions in posttest scores for participants with lower initial DCCS scores. 
In contrast, the hot condition had slightly greater posttest DCCS scores compared to the cool 
condition for participants with higher initial DCCS scores. Finally, a significant three-way 
interaction was found between condition, age and pre-DCCS, F(1, 225) = 4.09, p = .04, partial 
eta squared = .02, indicating that the age by condition interaction was affected by participants 


pre-DCCS score. 
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Figure 4. Posttest DCCS Scores in Relation to Pretest DCCS Scores for Hot and Cool AYCET 
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Analyses of Game Data 

Computer log data of the player’s interactions with the program while playing the game 
were recorded and analyzed. The computer log data was used to compare performance within the 
context of the game between the two conditions. Specific metrics included an accuracy measure 
of number of correct hits (i.e., number of “aliens saved”), and a speed measure of average 
reaction time. Independent t-tests conducted to determine if there were differences between the 
hot and cool conditions on the game metrics indicated no significant differences between 
conditions for the number of correct hits, (231) = 1.18, p = .24, or average reaction time, #(23/) 
= 1.57, p=.23. 

Relation of Game Log Data to Posttest DCCS Scores 

To determine if in-game performance was related to posttest score of shifting, 
correlations between the game performance metrics and DCCS posttest score calculated. 
Because there were no significant differences between conditions in the in-game metrics, the two 
conditions were considered together. Results from the analyses indicated that the correlation 
between the correct hits and the DCCS posttest score was significant, r = 0.24, p < 0.001. The 
correlation between average reaction time and DCCS posttest scores was also significant, r= - 
0.21, p =0.001. 

It may be that higher preexisting levels of EF allow participants to perform well on the 
AYCET game and also predict higher scores on the DCCS posttest. To account for this, partial 
correlations were conducted. Results from the partial correlations indicated that controlling for 
the effects of pretest DCCS, there is still a significant relation between the number of correct in- 


game hits and the DCCS posttest, r = 0.20, p = 0.002. The correlation between average reaction 
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time and DCCS posttest scores when controlling for pretest DCCS score was also significant, r= 


-0.13, p = 0.049. 


Discussion 
The goal of the current study was to investigate how developmental theory can inform the 


design of digital tools for learning and cognitive skills development. We examined implications 
of what is known about the neurophysiological and cognitive changes related to EF development 
during early to mid-adolescence, particularly developments in hot EF, for the design of a digital 
game to enhance the EF subskill of shifting. To do this, we used an emotional design approach 
(Plass et al., 2018; Um et al., 2012) to create two version of the EF training game, AYCET: one 
that was emotionally “hot”, and one that was emotionally “cool”. Overall, participants in both 
conditions demonstrated small, but significant improvements in the EF subskill of shifting (as 
measured by changes in the DCCS pre- to post-intervention), but the emotionally hot version of 
the game yielded significantly greater gains in EF. This was particularly true for the older 
adolescents in our study (ages 14 to 16 years), with little difference between the two conditions 
found for the younger adolescents (ages 12 to 13 years). Additionally, the hot condition was 
more effective for the adolescents with higher initial levels of EF, although this was a relatively 
small effect. Finally, a three-way effect between condition, age and prior EF suggests that both 
developmental and individual factors play a role in determining the most effective approach for 
training adolescents’ EF skills via digital games. 

These findings support our hypothesis that there would be significant gains in participants 
EF subskill of shifting, even after just 20-minutes of gameplay. They also supported our 
hypothesis that improvements in shifting would be greater for participants in the hot EF training 
group, particularly for the older adolescents in the study. These results are among the first to 


establish a connection between emotional design of a game and improvements of cognitive 
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skills. It should be noted that in general, the greatest improvements occur with the younger 
adolescents and the participants who had lower initial EF abilities. This is consistent with other 
studies on EF training, which have generally tended to be the most effective for individuals with 
the lowest EF at the beginning of the training (e.g., Homer, Ober, Rose, et al., 2018; Karbach & 
Kray, 2009; Holmes et al., 2009). 

Another set of questions involved the in-game metrics of accuracy (1.e., number of 
correct hits/aliens saved) and speed (1.e., average reaction time). As hypothesized, after 
accounting for pre-test DCCS score, both in-game metrics predicted posttest DCCS scores. This 
is consistent with the claim that the in-game activities of AYCET are supporting the 
development of the EF subskill of shifting. This approach of matching specific in-game activities 
to learning or developmental outcomes represents an effective way of validating the efficacy of 
specific game-based interventions. While randomized controlled trials remain the “gold 
standard” for determining whether or not an intervention is effective, game log data can help 
identify the specific in-game activities that are responsible for the desired outcomes, which can 
then inform the design of more effective games for learning. Contrary to our hypothesis, there 
were no significant differences between conditions on either of the in-game measures. The role 
of emotions in games for learning is often seen as solely enhancing engagement and motivation 
(Jabbar & Felicia, 2015). However, the finding of no between-group differences in the in-game 
metrics suggests that, in the current study, participants in the hot condition were not just more 
motivated and trying harder, as this would have resulted in better performance on the game. 

So, what exactly accounts for the effects of the hot condition of AYCET? The most likely 
factor is that increased emotional activation during play of the game is making the difference. 


Participants in the hot condition did report significantly more emotional engagement with the 
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game characters in AYCET, even though the effect was modest and was not significantly related 
to pre- to posttest gains in shifting. Additional research is needed to further explore this 
hypothesis, ideally collecting direct physiological measures during game-play to see if emotion 
response is predicting gains in EF. If increased engagement of emotion is making a difference, 
this would be consistent with the claim that tapping into adolescences development of hot EF can 
enhance the effects of interventions to improve EF. Perone et al. (2018) argue that the 
development of connections between the fronto-parietal and fronto-limbic networks during 
adolescence is involved in the development of hot EF. Although more data is needed to establish 
the neurophysiological underpinnings of hot EF, the timing does coincide with behavioral 
indications of hot EF, which show marked improvements in mid-adolescence, by 14 years of age 
(Prencipe et al., 2011). The developmental timing is consistent with the findings of the current 
study, in which the hot condition resulted in improve EF gains starting around 14 years of age. 
Interestingly, the effects seem to diminish by 15-years-old, although the efficacy of the 
emotional design approach for older adolescence should be investigated in future research. 

The findings from the current study contribute to earlier work examining the effects of 
emotional design on performance (Plass et al., 2014; Um et al., 2011), extending beyond this 
earlier work by examining the effects of emotional design in a game for improving cognitive 
skills in adolescents. Although adolescence can be a difficult developmental period to study 
(Davidson et al., 2006; Leshem, 2016), the current findings indicate a promising trend for 
creating more efficacious design features that may harness some of aspects of neurocognitive 


changes that occur during this important period of development. 


Limitations and Future Directions 
There are some limitations of the current study that should be addressed in future work. 


First, although significant pre- to posttest gains in shifting were found, given the brevity of the 
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current intervention (i.e., 20 minutes), the permanence of the gains is not clear. The goal of the 
current study was to examine the viability of using an emotional design approach for game-based 
EF interventions for adolescents. Given the promising results of the current study, next steps 
should include a longer intervention in a study that includes a control group in order to provide 
clearer evidence of the efficacy of the emotional design approach with digital games to make 
lasting improvements in adolescents’ EF. Prior research has found significant gains in EF with 
adolescents using versions of AYCET (e.g., Homer, Plass, at al., 2018; Parong et al., 2017), but 
to see if the hot version of the game is having a significant, lasting effect, future studies should 
compare the cool and hot EF versions over a longer training regimen. 

Secondly, measures of hot EF were not included in the current study. The study focused 
on the EF subskill of shifting, asking if a version of AYCET that was more emotionally engaging 
could enhance training of shifting in adolescence. However, additional work should determine 
both if adolescences’ prior level of hot EF influences the effects of the emotional design of the 
game, and if adolescents experienced improvements on hot EF as a result of playing AYCET. It 
seems possible that for the older adolescents, there may be greater gains in hot EF given that this 
is an essential development at that age. Furthermore, gains in hot EF may contribute adolescents’ 
ability to regulate emotions and may explain variability in emotional regulation (Woltering, 
Lishak, Hodgson, Granic, & Zelazo, 2016). Therefore, future work should examine not only 
gains in measures of cool and hot EF, but also possible benefits for crucial related skills, such as 
emotional regulations, that develop during adolescence. 

Related to this, more fundamental work is needed to understand further the relation 
between hot and cool EF, and how the nature of this interaction changes over the course of 


children and adolescent development. Neurophysiological research is needed to examine the 
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specific systems involved. For example, did older adolescent in the current study recruit more 
from regions in the PFC along with regions associated with reflective processing? Future studies 
examining the neural underpinnings associated with cognitive training success in an emotional 
design environment and how these change across development are warranted (Perone et al., 
2018). 

Another issue is that the cool version of AYCET was not actually all that “cool” and the 
“hot” version was not all that “hot”. A number of features in the cool version included 
emotionally relevant stimuli. For example, when a player made a correct hit in the cool version, 
the alien’s expression changed from neutral to a smile. The time pressure and points awarded for 
saving aliens was the same in both conditions, and very likely evoked some emotions. A truly 
neutral or cool version would not have included any type of emotional valence. With that said, 
the change in expression in the cool version was necessary as it served as feedback for the 
learner — and helped make the game enjoyable. Similarly, the hot version could be even “hotter”. 
For example, the game could have had even more emotionally salient stimuli or reward 
mechanisms (e.g., by having and engaging narrative or greater rewards and risks). Other game 
features, such as sounds and musical score, are known to affect player’s emotional response, and 
could be used to make the game even “hotter”. In ongoing research, we are also examining the 
use of virtual reality as a way of creating a game that is highly emotionally engaging — although 
this brings up the possibility that there may be point at which the emotional engagement of the 
game could be too great, and actually have a negative effect on learning outcomes. It seems 
likely that an “emotional sweet spot” for these types of interventions may exist, that will depend 


not only on the specific content of the games, but also on the developmental level of the 
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participants as well as individual variability in specific skills and abilities (e.g., level of hot and 
cool EF, emotional regulation, etc.). 

This relates to another area for future work, which involves adaptivity and 
personalization of games for learning and cognitive training. The most common way of adapting 
is to vary the difficulty of a game. Constantly challenge the learner has been shown to be an 
effective way of improving cognitive skills (Diamond & Lee, 2011). Adaptive algorithms, which 
monitor players’ performance and change the difficulty of the game accordingly, have been 
proven to be effect ways to keep players challenged at just above their current level of 
performance (Ericsson, 2006; Klingberg, 2010; von Bastian & Eschen, 2016). In the current 
study, both version of AYCET incorporated an adaptive algorithm that adapted challenge level 
based on in-game performance (1.e., speeding up delivery of aliens if the player was doing well, 
slowing it down if they were having problems). The results of this study suggest that, in addition 
of performance, developmental stage and individual variation in emotional response may be 


other factors that could inform the adaptive design of games for learning. 


Conclusion 
In the current study, an emotional design approach was used to activate hot EF in 


adolescents playing AYCET, a digital game designed to train the EF subskill of shifting. In 
general, the hot condition of the game, which was designed to be more emotionally activating, 
was more effective than the cool condition, particularly for the older adolescents and for 
adolescents with higher initial levels of EF. Overall, these results provide evidence for effective 
integration of emotional design into interventions for cognitive interventions. Furthermore, the 
results provide an example of how developmental theory can inform the design of effective 
games for learning and cognitive skills improvement. The results of the current study also have 


implications for our understanding of the developmental of EF, specifically supporting the notion 
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that hot and cool EF are not separate, but rather are part of an integrated system, with 
developments in hot EF lagging behind cool EF (Perone et al., 2018; Prencipe et al., 2011). 
Although additional research is needed to understand the developmental trajectory and 
neurophysiological underpinnings of hot and cool EF, as well as to explore the best ways to use 
emotional design to active hot EF in adolescents, the current study begins to provide some 


insight for understanding of both theoretical and applied issues involved in EF. 
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